What are differences between President from the South and North before civilwar?

In this project, we want to explore the difference between the president that is raised and born in the South(state with slavery) and North, Since after the civil war the cultural boundary between the North and South have been diminished. I just choose presidents from George Washington to Benjamin Harrison, the last president who has became adult before civil war,based on the assumptions that peoples ideology and characteristic are formed in their early adulthood so they should not be influenced by post-war situitions

packages.used=c("rvest", "tibble", "qdap", 
                "sentimentr", "gplots", "dplyr",
                "tm", "syuzhet", "factoextra", 
                "beeswarm", "scales", "RColorBrewer",
                "RANN", "tm", "topicmodels")
# check packages that need to be installed.
packages.needed=setdiff(packages.used, 
                        intersect(installed.packages()[,1], 
                                  packages.used))
# install additional packages
if(length(packages.needed)>0){
  install.packages(packages.needed, dependencies = TRUE)
}
# load packages
library("rvest")
library("tibble")
library("qdap")
library("sentimentr")
library("gplots")
library("dplyr")
library("tm")
library("png")
library("syuzhet")
library("factoextra")
library("beeswarm")
library("scales")
library("RColorBrewer")
library("RANN")
library("tm")
library("topicmodels")
library("xlsx")
library("wordcloud")
library("tidytext")
source("../lib/plotstacked.R")
source("../lib/speechFuncs.R")

This notebook was prepared with the following environmental settings.

print(R.version)

               _                           
platform       x86_64-apple-darwin15.6.0   
arch           x86_64                      
os             darwin15.6.0                
system         x86_64, darwin15.6.0        
status                                     
major          3                           
minor          4.3                         
year           2017                        
month          11                          
day            30                          
svn rev        73796                       
language       R                           
version.string R version 3.4.3 (2017-11-30)
nickname       Kite-Eating Tree            

Data Harvest

we selected all inaugural addresses of past presidents, for lack of data of nominal and farewell addresses of president before civil war.

### Inauguaral speeches
main.page <- read_html(x = "http://www.presidency.ucsb.edu/inaugurals.php")
# Get link URLs
# f.speechlinks is a function for extracting links from the list of speeches. 
inaug=f.speechlinks(main.page)
#head(inaug)
#as.Date(inaug[,1], format="%B %e, %Y")
inaug=inaug[-nrow(inaug),] # remove the last line, irrelevant due to error.

Using speech metadata posted on http://www.presidency.ucsb.edu/, We prepared xlsx data sets for the speeches we will scrap.

picture <- readPNG("map.png")
grid::grid.raster(picture)
#plot(picture)

This is the data I have processed. I just choose presidents from George Washington to Benjamin Harrison, the last president who has became adult before civil war, based on the assumptions that peoples ideology and characteristic are formed in their early adulthood so they should not be influenced by post-war new ideology and impacts.

About one-third presidents were born and raised in the South, while the others were raised in the North part of America. We use the common Mason–Dixon line to seperate south and north as most people do. In the South, most Presidents are from Virginia and Tennessee, in the North there are Presidents from New England, New York , iIlinois etc

inaug.list = read.xlsx("../data/inauglist_place.xlsx", sheetIndex = 1)

We assemble all scrapped speeches into one list. Note here that we dont have the full text yet, only the links to full text transcripts. scrap the texts of speeches from the speech URLs.

inaug.list$type=c(rep("inaug", nrow(inaug.list)))
inaug.list=cbind(inaug.list,inaug)

Based on the list of speeches, we scrap the main text part of the transcript’s html page. For simple html pages of this kind. For reproducibility, we also save our scrapped speeches into our local folder as individual speech files.

Now we combine the information from the internet and dataset, combine the columns of important data frame to generate further useful information, first combine the html and data and then download scripts to local repository.

# Loop over each row in speech.list
inaug.list$fulltext=NA
for(i in seq(nrow(inaug.list))) {
  text <- read_html(inaug.list$urls[i]) %>% # load the page
    html_nodes(".displaytext") %>% # isloate the text
    html_text() # get the text
inaug.list$fulltext[i]=text
  # Create the file name
 #filename <- paste0("../data/inaugurals/", 
#if(inaug.list$File[i]%in% nouu)#levels(sentencep.list.place$File))
#{
  filename <- paste0("../data/inaugurals/", 
                     inaug.list$type[i],
                     inaug.list$File[i], "-", 
                     inaug.list$Term[i], ".txt")
  sink(file = filename) %>% # open file to write 
    cat(text)  # write the file
  sink() # close the file
#}
}

We will use sentences as units of analysis for this project, as sentences are natural languge units for organizing thoughts and ideas. For each extracted sentence, we apply sentiment analysis using NRC sentiment. “The NRC Emotion Lexicon is a list of English words and their associations with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive). The annotations were manually done by crowdsourcing.”

We assign an sequential id to each sentence in a speech (sent.id) and also calculated the number of words in each sentence as sentence length (word.count).

sentence.list=NULL
for(i in 1:nrow(inaug.list)){
  sentences=sent_detect(inaug.list$fulltext[i],
                        endmarks = c("?", ".", "!", "|",";"))
  if(length(sentences)>0){
    emotions=get_nrc_sentiment(sentences)
    word.count=word_count(sentences)
    # colnames(emotions)=paste0("emo.", colnames(emotions))
    # in case the word counts are zeros?
    emotions=diag(1/(word.count+0.01))%*%as.matrix(emotions)
    sentence.list=rbind(sentence.list, 
                        cbind(inaug.list[i,-ncol(inaug.list)],
                              sentences=as.character(sentences), 
                              word.count,
                              emotions,
                              sent.id=1:length(sentences)
                        )
    )
  }
}

Some non-sentences exist in raw data due to erroneous extra end-of-sentence marks. And we also delete presidents that do not belong to our Category.

sentence.list=sentence.list%>%filter(!is.na(word.count)) 
sentence.list.place<-sentence.list[as.logical(1-is.na(sentence.list$Places)),]

Data analysis — length of sentences

For simpler visualization, we chose a subset of better known presidents or presidential candidates on which to focus our analysis. Let us haveOverview of sentence length distribution by different types of speeches. let us see how presidents inaugural length from different regions differ

inaugural speeches

par(mar=c(4, 11, 2, 2))
sentence.list.place$File=factor(sentence.list.place$File)
sentence.list.place$FileOrdered=reorder(sentence.list.place$File, 
                                      sentence.list.place$word.count, 
                                      mean, 
                                      order=T)
# this is for each president
sentence.list.place$Filedirection=paste(sentence.list.place$FileOrdered,sentence.list.place$Places)
beeswarm(word.count~Filedirection, 
         data=sentence.list.place,
         horizontal = TRUE, 
         pch=16, 
         col=alpha(brewer.pal(9, "Set1"), 0.6), 
         cex=0.55, cex.axis=0.8, cex.lab=0.8,
         spacing=5/nlevels(sentence.list.place$FileOrdered),
         las=2, xlab="Number of words in a sentence.", ylab="",
         main="Inaugural speeches")

# this is for overall places
beeswarm(word.count~Places,
         data=sentence.list.place,
         horizontal = TRUE, 
         pch=16, 
         col=alpha(brewer.pal(9, "Set1"), 0.6), 
         cex=0.55, cex.axis=0.8, cex.lab=0.8,
         spacing=5/nlevels(sentence.list.place$FileOrdered),
         las=2, xlab="Number of words in a sentence.", ylab="",
         main="Inaugural speeches")

And we want explore the short sentences presidents usually say:

#president from South
sentence.list.place%>%
  filter(File=="GeorgeWashington",word.count<=10)%>%
  select(sentences)#%>%sample_n(10)

sentence.list.place%>%
  filter(File=="JohnAdams", 
         word.count<=10)%>%
  select(sentences)

sentence.list.place%>%
  filter(File=="AbrahamLincoln", 
         word.count<=10)%>%
  select(sentences)#%>%sample_n(10)

#
folder.path="../data/inaugurals/"
ff.all<-Corpus(DirSource(folder.path))
ff.all<-tm_map(ff.all, stripWhitespace)
ff.all<-tm_map(ff.all, content_transformer(tolower))
ff.all<-tm_map(ff.all, removeWords, stopwords("english"))
ff.all<-tm_map(ff.all, removeWords, character(0))
ff.all<-tm_map(ff.all, removePunctuation)
tdm.all<-TermDocumentMatrix(ff.all)
tdm.tidy=tidy(tdm.all)
tdm.overall=summarise(group_by(tdm.tidy, term), sum(count))
wordcloud(tdm.overall$term, tdm.overall$`sum(count)`,
          scale=c(5,0.5),
          max.words=100,
          min.freq=1,
          random.order=FALSE,
          rot.per=0.3,
          use.r.layout=T,
          random.color=FALSE,
          #colors="BLUE"))
          colors=brewer.pal(9,"Blues"))

# the 
folder.path1="../data/inaugurals/beforewar/"
ff.beforewar<-Corpus(DirSource(folder.path1))
ff.beforewar<-tm_map(ff.beforewar, stripWhitespace)
ff.beforewar<-tm_map(ff.beforewar, content_transformer(tolower))
ff.beforewar<-tm_map(ff.beforewar, removeWords, stopwords("english"))
ff.beforewar<-tm_map(ff.beforewar, removeWords, character(0))
ff.beforewar<-tm_map(ff.beforewar, removePunctuation)
tdm.allbef<-TermDocumentMatrix(ff.beforewar)
tdm.tidybef=tidy(tdm.allbef)
tdm.overallb=summarise(group_by(tdm.tidybef, term), sum(count))
wordcloud(tdm.overallb$term, tdm.overallb$`sum(count)`,
          scale=c(5,0.5),
          max.words=100,
          min.freq=1,
          random.order=FALSE,
          rot.per=0.3,
          use.r.layout=T,
          random.color=FALSE,
          #colors="BLUE"))
          colors=brewer.pal(9,"Greens"))

folder.pathn="../data/inaugurals/beforewar/north"
ff.alln<-Corpus(DirSource(folder.pathn))
ff.alln<-tm_map(ff.alln, stripWhitespace)
ff.alln<-tm_map(ff.alln, content_transformer(tolower))
ff.alln<-tm_map(ff.alln, removeWords, stopwords("english"))
ff.alln<-tm_map(ff.alln, removeWords, character(0))
ff.alln<-tm_map(ff.alln, removePunctuation)
tdm.alln<-TermDocumentMatrix(ff.alln)
tdm.tidyn=tidy(tdm.alln)
tdm.overalln=summarise(group_by(tdm.tidyn, term), sum(count))
wordcloud(tdm.overalln$term,tdm.overalln$`sum(count)`,
          scale=c(5,0.5),
          max.words=100,
          min.freq=1,
          random.order=FALSE,
          rot.per=0.3,
          use.r.layout=T,
          random.color=FALSE,
          #colors="BLUE"))
          colors=brewer.pal(9,"Reds"))

folder.paths="../data/inaugurals/beforewar/south"
ff.alls<-Corpus(DirSource(folder.paths))
ff.alls<-tm_map(ff.alls, stripWhitespace)
ff.alls<-tm_map(ff.alls, content_transformer(tolower))
ff.alls<-tm_map(ff.alls, removeWords, stopwords("english"))
ff.alls<-tm_map(ff.alls, removeWords, character(0))
ff.alls<-tm_map(ff.alls, removePunctuation)
tdm.alls<-TermDocumentMatrix(ff.alls)
tdm.tidys=tidy(tdm.alls)
tdm.overalls=summarise(group_by(tdm.tidys, term), sum(count))
wordcloud(tdm.overalls$term, tdm.overalls$`sum(count)`,
          scale=c(5,0.5),
          max.words=100,
          min.freq=1,
          random.order=FALSE,
          rot.per=0.3,
          use.r.layout=T,
          random.color=FALSE,
          #colors="BLUE"))
          colors=brewer.pal(9,"Blues"))

Data analysis

Sentence length variation over the course of the speech, with emotions.

How our presidents (or candidates) alternate between long and short sentences and how they shift between different sentiments in their speeches. It is interesting to note that some presidential candidates’ speech are more colorful than others.

par(mfrow=c(4,1), mar=c(1,0,2,0), bty="n", xaxt="n", yaxt="n", font.main=1)
f.plotsent.len(In.list=sentence.list, InFile="GeorgeWashington", 
InType="inaug", InTerm=1, President="George Washington")
f.plotsent.len(In.list=sentence.list, InFile="ThomasJefferson", 
InType="inaug", InTerm=1, President="Thomas Jefferson")

f.plotsent.len(In.list=sentence.list, InFile="AbrahamLincoln", 
InType="inaug", InTerm=1, President="Abraham Lincoln")
f.plotsent.len(In.list=sentence.list, InFile="RutherfordBHayes", 
InType="inaug", InTerm=1, President="Rutherford B. Hayes")

Then I am going to find some sentences that can have a good represent of the president’s characteristic

cat("\nGeorge Washingto\n")

George Washingto

speech.df=tbl_df(sentence.list.place)%>%
filter(File=="GeorgeWashington", type=="inaug", word.count>=4)%>%
select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "Previous to the execution of any official act of the President the Constitution requires an oath of office."                                                                                                                                                         
[2] "and since the preservation of the sacred fire of liberty and the destiny of the republican model of government are justly considered, perhaps, as deeply, as finally, staked on the experiment entrusted to the hands of the American people."                       
[3] "between the genuine maxims of an honest and magnanimous policy and the solid rewards of public prosperity and felicity;"                                                                                                                                             
[4] "Previous to the execution of any official act of the President the Constitution requires an oath of office."                                                                                                                                                         
[5] "I dwell on this prospect with every satisfaction which an ardent love for my country can inspire, since there is no truth more thoroughly established than that there exists in the economy and course of nature an indissoluble union between virtue and happiness;"
[6] "From this resolution I have in no instance departed;"                                                                                                                                                                                                                
[7] "Having thus imparted to you my sentiments as they have been awakened by the occasion which brings us together, I shall take my present leave;"                                                                                                                       
[8] "Previous to the execution of any official act of the President the Constitution requires an oath of office."                                                                                                                                                         

cat("\nThomas Jefferson\n")

Thomas Jefferson

speech.df=tbl_df(sentence.list.place)%>%
filter(File=="ThomasJefferson", type=="inaug", Term==1, word.count>=5)%>%
select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "the supremacy of the civil over the military authority;"                                 
[2] "economy in the public expense, that labor may be lightly burthened;"                     
[3] "peace, commerce, and honest friendship with all nations, entangling alliances with none;"
[4] "the supremacy of the civil over the military authority;"                                 
[5] "peace, commerce, and honest friendship with all nations, entangling alliances with none;"
[6] "peace, commerce, and honest friendship with all nations, entangling alliances with none;"
[7] "the supremacy of the civil over the military authority;"                                 
[8] "peace, commerce, and honest friendship with all nations, entangling alliances with none;"

cat("\nAbraham Lincoln\n")

Abraham Lincoln

speech.df=tbl_df(sentence.list.place)%>%
filter(File=="AbrahamLincoln", type=="inaug", Term==1, word.count>=4)%>%
select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "Whoever rejects it does of necessity fly to anarchy or to despotism."                        
[2] "In our present differences, is either party without faith of being in the right?"            
[3] "'May' Congress prohibit slavery in the Territories?"                                         
[4] "The Government will not assail 'you'."                                                       
[5] "Though passion may have strained it must not break our bonds of affection."                  
[6] "Whoever rejects it does of necessity fly to anarchy or to despotism."                        
[7] "The Government will not assail 'you'."                                                       
[8] "Perpetuity is implied, if not expressed, in the fundamental law of all national governments."

cat("\nRutherford B. Hayes\n")

Rutherford B. Hayes

speech.df=tbl_df(sentence.list.place)%>%
filter(File=="RutherfordBHayes", type=="inaug", Term==1, word.count>=5)%>%
select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "Many of the calamitous efforts of the tremendous revolution which has passed over the Southern States still remain."                                                                                                                                                                                                                                                                                 
[2] "They neither expected nor desired from public officers any partisan service."                                                                                                                                                                                                                                                                                                                        
[3] "That tribunal--established by law for this sole purpose;"                                                                                                                                                                                                                                                                                                                                            
[4] "That tribunal--established by law for this sole purpose;"                                                                                                                                                                                                                                                                                                                                            
[5] "The circumstances were, perhaps, in no respect extraordinary save in the closeness and the consequent uncertainty of the result."                                                                                                                                                                                                                                                                    
[6] "That it was a wise, just, and providential act, fraught with good for all concerned, is not generally conceded throughout the country."                                                                                                                                                                                                                                                              
[7] "If, unhappily, questions of difference should at any time during the period of my Administration arise between the United States and any foreign government, it will certainly be my disposition and my hope to aid in their settlement in the same peaceful and honorable way, thus securing to our country the great blessings of peace and mutual good offices with all the nations of the world."
[8] "That tribunal--established by law for this sole purpose;"                                                                                                                                                                                                                                                                                                                                            

Clustering of emotions

heatmap.2(cor(sentence.list%>%filter(type=="inaug")%>%select(anger:trust)), 
scale = "none", 
col = bluered(100), , margin=c(6, 6), key=F,
trace = "none", density.info = "none")
par(mar=c(4, 6, 2, 1))

emo.means.north=colMeans(select(sentence.list.place%>%filter(Places=="N"), anger:trust)>0.01)
emo.means.south=colMeans(select(sentence.list.place%>%filter(Places=="S"), anger:trust)>0.01)
col.use=c("red2", "darkgoldenrod1", 
"chartreuse3", "blueviolet",
"darkgoldenrod2", "dodgerblue3", 
"darkgoldenrod1", "darkgoldenrod1")
par(mfrow=c(1,2))

barplot(emo.means.north[order(emo.means.north)], las=2, col=col.use[order(emo.means.north)], horiz=T, main="North likely")
barplot(emo.means.south[order(emo.means.south)], las=2, col=col.use[order(emo.means.south)], horiz=T, main="South likely")

presid.summary=tbl_df(sentence.list.place)%>%
filter(type=="inaug",)%>%
#group_by(paste0(type, File))%>%
group_by(Filedirection)%>%
summarise(
anger=mean(anger),
anticipation=mean(anticipation),
disgust=mean(disgust),
fear=mean(fear),
joy=mean(joy),
sadness=mean(sadness),
surprise=mean(surprise),
trust=mean(trust)
#negative=mean(negative),
#positive=mean(positive)
)
presid.summary=as.data.frame(presid.summary)
rownames(presid.summary)=as.character((presid.summary[,1]))
km.res=kmeans(presid.summary[,-1], iter.max=200,
5)
fviz_cluster(km.res, 
stand=F, repel= TRUE,
data = presid.summary[,-1], xlab="", xaxt="n",
show.clust.cent=FALSE)

LDA Data analysis

For topic modeling, we prepare a corpus of sentence snipets as follows. For each speech, we start with sentences and prepare a snipet with a given sentence with the flanking sentences.

sentence.list<-sentence.list.place
corpus.list=sentence.list[2:(nrow(sentence.list)-1), ]
sentence.pre=sentence.list$sentences[1:(nrow(sentence.list)-2)]
sentence.post=sentence.list$sentences[3:(nrow(sentence.list)-1)]
corpus.list$snipets=paste(sentence.pre, corpus.list$sentences, sentence.post, sep=" ")
rm.rows=(1:nrow(corpus.list))[corpus.list$sent.id==1]
rm.rows=c(rm.rows, rm.rows-1)
corpus.list=corpus.list[-rm.rows, ]

Text mining

docs <- Corpus(VectorSource(corpus.list$snipets))
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

Here too experience has afforded us the most satisfactory proof in its favor. Just as this Constitution was put into action several of the principal States of Europe had become much agitated and some of them seriously convulsed. Destructive wars ensued, which have of late only been terminated.

#remove potentially problematic symbols
docs <-tm_map(docs,content_transformer(tolower))
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

the sagacious mind of mr. jefferson early saw and lamented this error, and attempts have been made, hitherto without success, to apply the amendatory power of the states to its correction. as, however, one mode of correction is in the power of every president, and consequently in mine, it would be useless, and perhaps invidious, to enumerate the evils of which, in the opinion of many of our fellow-citizens, this error of the sages who framed the constitution may have been the source and the bitter fruits which we are still to gather from it if it continues to disfigure our system.

#remove punctuation
docs <- tm_map(docs, removePunctuation)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

others in my judgment are attributable to a misconstruction of some of its provisions of the former is the eligibility of the same individual to a second term of the presidency the sagacious mind of mr

#Strip digits
docs <- tm_map(docs, removeNumbers)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

in justly balancing the powers of the federal and state authorities difficulties nearly insurmountable arose at the outset and subsequent collisions were deemed inevitable amid these it was scarcely believed possible that a scheme of government so complex in construction could remain uninjured from time to time embarrassments have certainly occurred

#remove stopwords
docs <- tm_map(docs, removeWords, stopwords("english"))
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

    highly favored condition   country    interest  every citizen  maintain     dangers  menace us   exist     ascertained  guarded 

#remove whitespace
docs <- tm_map(docs, stripWhitespace)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

 territory enlarged fair honorable treaty great advantage original states states respectively protected national government mild parental system foreign dangers enjoying within separate spheres wise partition power just proportion sovereignty improved police extended settlements attained strength maturity best proofs wholesome laws well administered look condition individuals proud spectacle exhibit

#Stem document
docs <- tm_map(docs,stemDocument)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

made recommend pledg public exig requir recurr time remain trust equal promptitud perform duti alik incumb experi now make will seen next session congress whether revenu shall augment adequ necessari purpos defici still continu especi probabl perman cours pursu appear obvious

Topic modeling

Gengerate document-term matrices.

dtm <- DocumentTermMatrix(docs)
#convert rownames to filenames#convert rownames to filenames
rownames(dtm) <- paste(corpus.list$type, corpus.list$File,
corpus.list$Term, corpus.list$sent.id, sep="_")
rowTotals <- apply(dtm , 1, sum) #Find the sum of words in each Document
dtm  <- dtm[rowTotals> 0, ]
corpus.list=corpus.list[rowTotals>0, ]

Run LDA

#Set parameters for Gibbs sampling
burnin <- 4000
iter <- 2000
thin <- 500
seed <-list(2003,5,63,100001,765)
nstart <- 5
best <- TRUE
#Number of topics
k <- 15
#Run LDA using Gibbs sampling
ldaOut <-LDA(dtm, k, method="Gibbs", control=list(nstart=nstart, seed = seed, best=best,burnin = burnin, iter = iter, thin=thin))
#write out results
#docs to topics
ldaOut.topics <- as.matrix(topics(ldaOut))
table(c(1:k, ldaOut.topics))

  1   2   3   4   5   6   7   8   9  10  11  12  13  14  15 
149 146 178 131 155 156 168 162 113 129 147 152 145 152 120 

write.csv(ldaOut.topics,
          file=paste("../out/LDAGibbs",k,"DocsToTopics.csv"))
#top 6 terms in each topic
ldaOut.terms <- as.matrix(terms(ldaOut,20))
write.csv(ldaOut.terms,
          file=paste("../out/LDAGibbs",k,"TopicsToTerms.csv"))
#probabilities associated with each topic assignment
topicProbabilities <- as.data.frame(ldaOut@gamma)
write.csv(topicProbabilities,file=paste("/Users/westspringamerica/Downloads/wk2-TextMining/out/LDAGibbs",k,"TopicProbabilities.csv"))

terms.beta=ldaOut@beta
terms.beta=scale(terms.beta)
topics.terms=NULL
for(i in 1:k){
topics.terms=rbind(topics.terms, ldaOut@terms[order(terms.beta[i,], decreasing = TRUE)[1:7]])
}
topics.terms

      [,1]       [,2]        [,3]         [,4]        [,5]         [,6]        [,7]           
 [1,] "knowledg" "appeal"    "district"   "insist"    "process"    "dream"     "alter"        
 [2,] "portion"  "industri"  "agricultur" "indian"    "tariff"     "discrimin" "foster"       
 [3,] "station"  "oath"      "presenc"    "obedi"     "overwhelm"  "dilig"     "shrink"       
 [4,] "path"     "preced"    "repeat"     "announc"   "occurr"     "weigh"     "seced"        
 [5,] "cours"    "friend"    "intercours" "allianc"   "negoti"     "diplomaci" "kindr"        
 [6,] "armi"     "invas"     "late"       "militia"   "thousand"   "battl"     "compel"       
 [7,] "unit"     "territori" "weak"       "dominion"  "everyth"    "western"   "discourag"    
 [8,] "sphere"   "endang"    "tend"       "discord"   "impli"      "contend"   "discountenanc"
 [9,] "appoint"  "reform"    "promis"     "partisan"  "conveni"    "compens"   "color"        
[10,] "intellig" "corrupt"   "broad"      "descend"   "comprehens" "victori"   "scope"        
[11,] "doubt"    "economi"   "soon"       "credit"    "contract"   "privat"    "payment"      
[12,] "decis"    "class"     "decid"      "imperfect" "obey"       "deliv"     "stake"        
[13,] "legisl"   "grant"     "major"      "earli"     "languag"    "conform"   "conceiv"      
[14,] "father"   "restor"    "god"        "fruit"     "mankind"    "fortun"    "ardent"       
[15,] "injuri"   "roman"     "resist"     "bosom"     "democraci"  "introduc"  "aristocraci"  

ldaOut.terms

      Topic 1    Topic 2      Topic 3         Topic 4     Topic 5         Topic 6     Topic 7     Topic 8     Topic 9    
 [1,] "right"    "interest"   "duti"          "will"      "nation"        "war"       "state"     "govern"    "shall"    
 [2,] "upon"     "protect"    "confid"        "principl"  "peac"          "made"      "unit"      "union"     "will"     
 [3,] "citizen"  "within"     "may"           "now"       "foreign"       "place"     "territori" "state"     "offic"    
 [4,] "equal"    "duti"       "trust"         "great"     "interest"      "without"   "great"     "constitut" "civil"    
 [5,] "everi"    "revenu"     "countri"       "import"    "justic"        "forc"      "increas"   "power"     "servic"   
 [6,] "american" "limit"      "fellowcitizen" "countri"   "respect"       "great"     "extend"    "general"   "polit"    
 [7,] "secur"    "make"       "best"          "upon"      "countri"       "necessari" "new"       "form"      "give"     
 [8,] "person"   "far"        "high"          "part"      "relat"         "sinc"      "mani"      "preserv"   "expect"   
 [9,] "without"  "portion"    "faith"         "may"       "polici"        "time"      "popul"     "exercis"   "administr"
[10,] "enjoy"    "industri"   "whose"         "year"      "power"         "purpos"    "independ"  "feder"     "effort"   
[11,] "condit"   "encourag"   "take"          "opinion"   "honor"         "defens"    "limit"     "object"    "public"   
[12,] "found"    "product"    "respons"       "parti"     "maintain"      "resourc"   "remain"    "institut"  "support"  
[13,] "possess"  "use"        "call"          "subject"   "just"          "progress"  "becom"     "author"    "influenc" 
[14,] "act"      "commerc"    "shall"         "conduct"   "cours"         "land"      "less"      "system"    "need"     
[15,] "can"      "home"       "feel"          "sentiment" "well"          "might"     "except"    "whole"     "purpos"   
[16,] "posit"    "manufactur" "discharg"      "elect"     "friend"        "event"     "whole"     "domest"    "best"     
[17,] "rest"     "countri"    "oblig"         "made"      "desir"         "suffici"   "commerc"   "concern"   "appoint"  
[18,] "privileg" "benefit"    "just"          "depend"    "intercours"    "militari"  "near"      "individu"  "reform"   
[19,] "suffer"   "find"       "sens"          "administr" "fellowcitizen" "work"      "vast"      "harmoni"   "proper"   
[20,] "advantag" "mean"       "success"       "accord"    "preserv"       "armi"      "enlarg"    "common"    "promot"   
      Topic 10     Topic 11     Topic 12   Topic 13     Topic 14    Topic 15  
 [1,] "peopl"      "public"     "law"      "constitut"  "happi"     "spirit"  
 [2,] "govern"     "will"       "one"      "power"      "hope"      "exist"   
 [3,] "can"        "may"        "can"      "execut"     "prosper"   "everi"   
 [4,] "free"       "secur"      "question" "congress"   "bless"     "peopl"   
 [5,] "nation"     "requir"     "differ"   "effect"     "futur"     "continu" 
 [6,] "human"      "measur"     "view"     "legisl"     "look"      "danger"  
 [7,] "present"    "demand"     "practic"  "depart"     "let"       "institut"
 [8,] "mind"       "debt"       "section"  "upon"       "liberti"   "time"    
 [9,] "hand"       "govern"     "must"     "grant"      "still"     "caus"    
[10,] "among"      "doubt"      "parti"    "control"    "even"      "liberti" 
[11,] "natur"      "experi"     "opinion"  "presid"     "world"     "result"  
[12,] "intellig"   "necessari"  "case"     "instrument" "year"      "patriot" 
[13,] "noth"       "faith"      "good"     "author"     "establish" "long"    
[14,] "repres"     "provid"     "may"      "might"      "patriot"   "consequ" 
[15,] "love"       "expenditur" "claim"    "act"        "fear"      "true"    
[16,] "order"      "money"      "other"    "given"      "present"   "republ"  
[17,] "enlighten"  "everi"      "anoth"    "major"      "father"    "chang"   
[18,] "virtu"      "return"     "believ"   "howev"      "man"       "system"  
[19,] "republican" "economi"    "decis"    "becom"      "countri"   "injuri"  
[20,] "men"        "due"        "provis"   "first"      "exampl"    "none"    

Based on the most popular terms and the most salient terms for each topic, we assign a hashtag to each topic. This part require manual setup as the topics are likely to change.

topics.oldhash=c("Economy", "America", "Defense", "Belief", "Election", "Patriotism", "Unity", "Government", "Reform", "Temporal", "WorkingFamilies", "Freedom", "Equality", "Misc", "Legislation")
topicsnew.hash=c("everi", "war" ,"law",  "upon","nation", "union", "constitut","will","public","interest","state",     "great","duti", "peopl","right")
topics.hash=topicsnew.hash
corpus.list$ldatopic=as.vector(ldaOut.topics)
corpus.list$ldahash=topics.hash[ldaOut.topics]
colnames(topicProbabilities)=topics.hash
corpus.list.df=cbind(corpus.list, topicProbabilities)

Clustering of topics

par(mar=c(1,1,1,1))
topic.summary=tbl_df(corpus.list.df)%>%
filter(type%in%c("inaug"))%>%
select(Filedirection, everi: right)%>%
group_by(Filedirection)%>%
summarise_each(funs(mean))

`summarise_each()` is deprecated.
Use `summarise_all()`, `summarise_at()` or `summarise_if()` instead.
To map `funs` over all variables, use `summarise_all()`

topic.summary=as.data.frame(topic.summary)
rownames(topic.summary)=topic.summary[,1]
topic.plot=c(1:15)
print(topics.hash[topic.plot])

 [1] "everi"     "war"       "law"       "upon"      "nation"    "union"     "constitut" "will"      "public"    "interest" 
[11] "state"     "great"     "duti"      "peopl"     "right"    

heatmap.2(as.matrix(topic.summary[,topic.plot+1]), 
scale = "column", key=F, 
col = bluered(100),
cexRow = 0.8, cexCol = 0.9, margins = c(8, 8),
trace = "none", density.info = "none")

    
par(mfrow=c(5, 1), mar=c(1,1,2,0), bty="n", xaxt="n", yaxt="n")
topic.plot=c(1, 13, 14, 15, 8, 9, 12)
print(topics.hash[topic.plot])

[1] "everi"  "duti"   "peopl"  "right"  "will"   "public" "great" 

speech.df=tbl_df(corpus.list.df)%>%filter(File=="GeorgeWashington", type=="inaug",Term==1)%>%select(sent.id, everi: right)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1], 
xlab="Sentences", ylab="Topic share", main="GeorgeWashington")

[1] 0.05298456 0.10596912 0.15895368 0.21193824 0.26492280 0.31790736 0.37089192

speech.df=tbl_df(corpus.list.df)%>%filter(File=="ThomasJefferson", type=="inaug", Term==1)%>%select(sent.id, everi: right)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
xlab="Sentences", ylab="Topic share", main="ThomasJefferson")

[1] 0.06071925 0.12143850 0.18215775 0.24287700 0.30359625 0.36431550 0.42503475

speech.df=tbl_df(corpus.list.df)%>%filter(File=="UlyssesSGrant", type=="inaug", Term==1)%>%select(sent.id, everi: right)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1], 
xlab="Sentences", ylab="Topic share", main="UlyssesSGrant")

[1] 0.05193823 0.10387646 0.15581469 0.20775292 0.25969115 0.31162938 0.36356761

speech.df=tbl_df(corpus.list.df)%>%filter(File=="AbrahamLincoln", type=="inaug", Term==1)%>%select(sent.id, everi: right)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
xlab="Sentences", ylab="Topic share", main="AbrahamLincoln")

[1] 0.06509839 0.13019677 0.19529516 0.26039354 0.32549193 0.39059031 0.45568870

speech.df=tbl_df(corpus.list.df)%>%filter(File=="JamesMonroe", type=="inaug",Term==1)%>%select(sent.id, everi: right)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
xlab="Sentences", ylab="Topic share", main="JamesMonroe")

[1] 0.05186465 0.10372929 0.15559394 0.20745859 0.25932323 0.31118788 0.36305253

presid.summary=tbl_df(corpus.list.df)%>%
filter(type=="inaug")%>%
select(Filedirection, everi: right)%>%
group_by(Filedirection)%>%
summarise_each(funs(mean))

`summarise_each()` is deprecated.
Use `summarise_all()`, `summarise_at()` or `summarise_if()` instead.
To map `funs` over all variables, use `summarise_all()`

presid.summary=as.data.frame(presid.summary)
rownames(presid.summary)=as.character((presid.summary[,1]))
km.res=kmeans(scale(presid.summary[,-1]), iter.max=200,
5)
fviz_cluster(km.res, 
stand=T, repel= TRUE,
data = presid.summary[,-1],
show.clust.cent=FALSE)

Conclusions:

According to the analysis, we find that

When it comes to number of words in presidents’augural presentation, the south is not statistically different from the north, while they do speak a little shorter on average.

All presidents like to use the word “WIll”,maybe they are promising the audience with a good future or they would like to show his strong willingness. In early period “People” and “Government” has more chance to appear in the presentation than nowadays. Presidents from north like to say about “People”, while “South” is more likely to talk about States. This really make sense that Southerners want a more united States instead of a United States with a strong government. And the north people urge them to liberate people of colored races.

When evaluating emotions and length of Presidents,President lincoln’s passage are very colorful of rhythm and emotions, we know he is very good at giving speech. As a whole, northern presidents has more fear and sad feelings then the south,while the south are more joyful and likely to surpise, this may to due to coldness of the north making people sorrow while the south has more sunny weathers. the cluster is not good enough to give the overview of all presidents because each president has his own unique features

By topic modelling we find that the old topic are quite different from newer ones.and the heatmap are more clear to show there are southern and northern presidents seem to have some inner cores ,they may the same ancient synonym ancestry. and we also find the topics they are talking about are quite the same but not in the same form